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(57) Abstract 

A trellis coded modulation system comprises a source of successive 2-bit data symbols Xi, Xj arranged in a frame format wherein each 
frame comprises a pluraUty of data segments each including a plurality of groups of interieaved data symbols. Each group of interieaved 
data symbols is separately coded by a precoder (32a) and convolution encoder (32b) to derive coded ouqjut symbols Zo, Zi, Z2, which are 
mapped to respective 8-ievel symbols for transmission together with periodically generated frame and segment sync symbols. The received 
signal may be filtered by a linear filter (42), e.g. a comb filter (42), to reduce co-channel interference and each group of filtered symbols 
is applied to a respective first Viterbi decoder (44) for estimating data bits Xi, X2. Each first decoder (44) preferably comprises a reduced 
complexity Viterbi decoder (44) responsive to a partial representation of the state of the linear filter (42). Each group of received symbols 
may also be directly applied to a respective second Viterbi decoder (46) for estimating data bits Xi X2. Estimated data bits Xi X2 from the 
first or second decoders (44 or 46) are selected for further processing. 
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TRELLIS CQDSD MODTJIATTON SYSTSM FOR WDTV 
'Backgiround of the Invention 

The present invention relates to trellis coded 
modulation (TCM) transmission and reception systems and 
particularly concerns the use of TCM in a high definition 
television (HDTV) application. 

Trellis coded modulation is a well known technique 
for improving the performance of digital transmission and 
reception systems. Improvements can be achieved in S/N 
performance at a given power level or alternatively, the 
transmitted power recjuired to achieve a given S/N 
performance can be reduced. In essence, TCM comprises 
the use of a multi- state convolution encoder to convert 
each k input data bits of an input sequence of data bits 
into k + n output bits, and is therefore referred to as a 
rate k/ (k + n) convolution encoder. The output bits are 
then mapped into a secpaence of discrete symbols (having 
2 (k+n) values) of a modulated carrier for data 
transmission. The symbols may, for example, comprise 
2 (k+n) phase or amplitude values. By coding the input 
data bits in a state -dependent sequential manner, 
increased minimum Euclidean distances between the 
allowable transmitted sequences may be achieved leading 
to a reduced error probability when a maximum 
likelihood decoder, (e.g. a Viterbi decoder) is used in 
the receiver. 

Figure 1 generally illustrates a system of the type 
described above. Each k bits of an input data stream is 
converted to k + n output bits by a rate k/ (k + n) state- 
dependent sequential convolution encoder 10. Each group 
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of (k + n) output bits is then mapped to one of 2 (5^+^) 
symbols by a mapper 12. The symbols are transmitted over 
a selected channel by a transmitter 14 . A receiver 
includes a tuner 16 for converting the signal received 
over the selected channel to an intermediate frequency 
signal, which is demodulated by a demodulator 18 to 
provide a baseband analog signal. The analog signal is 
appropriately sampled 

by an A/D 20 to recover the transmitted symbols which are 
then applied to a Viterbi decoder 22 for recovering the 
original k data bits. 

U. S. Patent No. 5,087,975 discloses a vestigial 
sideband (VSB) system for transmitting a television 
signal in the form of successive M-level symbols over a 
standard € MHz television channel . The symbol rate is 
preferably fixed at about 6 84H {about 10.76 Mega 
symbols/sec) , where H is the NTSC horizontal scanning 
frequency. This patent also discloses the use of a 
receiver comb filter having a feed forward delay of 12 
symbol clock intervals for reducing NTSC co- channel 
interference in the receiver. In order to facilitate 
operation of the receiver comb filter, the source data is 
preceded by a modulo- filter having a feedback delay of 12 
symbol clock intervals. In the receiver of the patented 
system a complementary modulo postcoder may be used to 
process the received signal in lieu of the comb filter 
in the absence of significant NTSC co-channel 
interference to avoid the degradation of S/N performance 
attributable thereto. 
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It is an object of the present invention to provide 
a digital transmission and reception system incorporating 
both TCM techniques and a receiver comb filter for 
achieving improved S/N performance with NTSC co- channel 
interference reduction. 

It is a further object. of the invention to provide a 
digital transmission and reception system of the 
foregoing type in which receiver complexity is reduced 
without significantly degrading performance. 

It is yet another object of the invention to provide 
a novel frame structure and synchronization system for a 
digital television signal. 
Brief Description q£ thg Prawingig 

These and other objects and advantages of the 
invention will be apparent upon reading the following 
description in conjunction with the drawings, in which: 

Figure 1 is a system block diagram of a conventional 
TCM system employing an optimal MLSE Viterbi decoder; 

Figure 2A is a system block diagram of a television 
signal transmitter and receiver including a TCM system 
employing Viterbi decoding according to the present 
inventions- 
Figure 2B is a block diagram of an alternate 
embodiment of the receiver of Figure 2A; 

Figure 3 illustrates the symbol interleaving 
effected in the transmitter of Figure 2; 

Figure 4 is a block diagram illustrating circuits 3 2 
and 3 4 of Figure 2 in more detail; 

Figure 5 is a diagram illustrating the operation of 
mapper 4 9 of Figure 4; 
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Figure 6 is a table illustrating the operation of 
convolution encoder 32b of Figure 4; 

Figure 7 is a trellis state transition diagram based 
upon the table of Figure 6; 

Figure 8 is a block diagram illustrating circuits 
42, 44, 46 and 48 of Figure 2 in more details- 
Figure 9 is a functional block diagram of optimal 
MLSE Viterbi decoders 4 6A-4 6L of Figure 8; 

Figure 10 is a diagram showing a circuit which may- 
be used in place of Viterbi decoders 46A-46L of Figure 8 
for recovering estimations of bits and ¥2 ; 

Figure 11 is a functional block diagram of optimal 
MLSE Viterbi decoders 44A-44L of Figure 8; 

Figure 12 is a table illustrating the operation of 
the TCM encoder of the invention including the effects 
introduced by comb filter 42 of the receiver of Figure 2; 

Figures 13 shows the resultant effect of combining 
two subsets in comb filter 42 and the resultant cosets 
that arise; 

Figure 14 shows the seven cosets that occur in the 
table of Figure 13; 

Figure 15 is a trellis state transition diagram 
based on the table of Figure 12 ; 

Figure 16 is a functional block diagram of a Viterbi 
decoder programmed on the basis of the trellis diagram of 
Figure 15; 

Figure 17 is a block diagram illustrating the use of 
the Viterbi decoder of Figure 16 to recover estimations 
of transmitted bits and X2 ; 
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Figure 18 illustrates the states of delay elements 

48, 54 and 56 of Figure 4 after a segment sync intervals- 
Figure 19 illustrates the format of the signal 

developed at the output of multiplexer 62 of Figure 4 in 

the vicinity of a segment sync signals- 
Figure 2 0 is a block diagram of comb filter 42 of 

Figure 8 modified for processing data segment and frame 

sync signals; 

Ficpire 21 is a block diagram of a postcoder 48A-48Ij 
of Figure 8 modified for processing data segment and 
frame sync signals; 

Figure 22 illustrates the format of the signal 
developed at the output of multiplexer 62 of Figure 4 in 
the vicinity of a frame sync signal; 

Figure 23 illustrates an embodiment of the invention 
in which an increased bit rate transmission is achieved 
by providing input data in the form of 3 bits per symbol; 

Figures 24A and 24B illustrate the application of 
the invention to a QAM system; and 

Figures 25A and 25B illustrate respective postcoder 
configurations useful in receivers for the embodiments of 
the invention shown in Figures 23 and 24. 
DescriTDtion of the Pref^rrfid Emhori■im^rt^« 

Figure 2A generally illustrates a TCM system applied 
to a multilevel VSB HDTV transmission and reception 
system of the type disclosed in the '975 patent. While 
the multilevel VSB HDTV application is contemplated in 
the preferred embodiment of the invention, it will be 
understood that the invention is more general in nature 
and thus may be applied to other types of transmission 
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and reception systems, including lower resolution video 
systems as well as non-video based data systems. Also, 
other modulation techniques, such as those employing, 
for example, quadrature amplitude modulation (QAM) may 
be employed. 

With further reference to Figure 2A, a data source 
24 provides a succession of data bytes which may, for 
example, comprise a compressed HDTV signal, a compressed 
television signal of NTSC resolution or any other digital 
data signal. The data bytes are preferably, although not 
necessarily, arranged in successive frames each 
including, on an alternating basis, 262 and 263 data 
segments, each data segment comprising 684 two-bit 
symbols occurring at a symbol rate of about 10.76 
Msymbols/sec. The data bytes from source 24, which also 
provides a plurality of timing signals, are applied to a 
Reed-Solomon encoder 26 for forward error correction 
coding and therefrom to a byte interleaver 28. Byte 
interleaver 28 reorders the data bytes throughout a frame 
to reduce the susceptibility of the system to burst 
noise . 

The interleaved data bytes from interleaver 28 are 
applied to a symbol interleaver 3 0 which provides in a 
preferred embodiment two output bit streams X^, X2 at the 
symbol rate, each bit pair Xi, X2 corresponding to a data 
symbol. In particular, due to the presence of the comb 
filter in the receiver (to be described in more detail 
hereinafter) , it is desirable to interleave the 2 -bit 
symbols of each data segment among 12 subsegments A-L, 
each comprising 57 symbols, as shown in Figure 3. Each 
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subsegment, e.g. subsegment A, thus comprises 57 symbols, 
e.g. Aq - ^sSf separated from each other by 12 symbol 
intervals. Symbol interleaver 30 effects such by 
reordering the applied 2 -bit symbols of each data byte as 
four successive symbols of a respective subsegment. 
Thus, for example, the four 2 -bit symbols of the first 
data byte applied to interleaver 3 0 are provided as 
output symbols Aq, A^, A2, and A3 of siibsegment A, the 
four 2 -bit symbols of the second applied data byte as 
output symbols Bq, B^, B2 and B3 of subsegment B, and so 
on. This insures that the symbols of each data byte are 
processed as a unit both in the encoder and the receiver. 

The stream of 2 -bit symbols from interleaver 3 0 are 
coupled to a precoder and trellis encoder 32 for 
conversion to 3 output bits as will be described in 
further detail hereinafter. Since unit 32 is 
characterized by a 12 -symbol delay, it may be thought of 
as comprising 12 parallel encoders each operating at 1/12 
the symbol clock rate, with each subsegment generated by 
interleaver 3 0 being processed by a respective one of the 
parallel encoders. The stream of 3 -bit symbols developed 
at the output of unit 32 is applied to a symbol mapper 
and sync inserter 34 and therefrom to a VSB modulator 36 
for transmission as a plurality of 8 -level symbols. 

The transmitted signal is received by a receiver 
including a tuner, demodulator and A/D 4 0 corresponding 
to blocks 16, 18 and 20 of Figure 1. The output of unit 
40 comprises a stream of multibit (e.g. 8-10 bit) 8-level 
symbols which are applied by components 50a, b, c, and d 
of a selector switch 50 (see U. S. Patent No. 5,260,793 
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for an exemplary embodiment of a circuit for operating 
switch 50) to a first processing path comprising a comb 
filter 42 and a first Viterbi decoder 44 and to a second 
processing path comprising a second Viterbi decoder 46 
and a postcoder 48. Each of the processing paths includes 
an equalizer 3 8 coupled between switching components 50b 
and 50c. The outputs of both Viterbi decoder 44 and 
postcoder 4 8 each comprise reconstructions of bit streams 
Xj, X2 . Component 50d of selector switch 50 couples one 
of the applied bit stream pairs , X2 to a symbol 
deinterleaver 52 which reconstructs the original data 
bytes. These data bytes are then deinterleaved by byte 
deinterleaver 54 and error- corrected by Reed-Solomon 
decoder 56 for application to the remainder of the 
receiver. 

An alternate embodiment of the receiver of Figure 2A 
is shown in Figure 2B. This embodiment is generally 
similar to the system of Figure 2A except that only one 
Viterbi decoder 45 is provided. More specifically, 
Viterbi decoder 45 is responsive to a control signal from 
selector switch 50 for assuming a first configuration for 
implementing the functions of Viterbi decoder 44 when the 
first processing path is selected and for assuming a 
second configuration for implementing the functions of 
Viterbi decoder 4 6 when the second processing path is 
selected , 

Referring to Figure 4, unit 32 comprises a modulo-2, 
feedback precoder 32a receiving the symbols (each symbol 
being identified as bits X^, X2) from interleaver 3 0 for 
developing output bits Y^, Y2 . More specifically. 
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precoder 32a comprises a modulo-2 summer 44 having a 
first input connected for receiving bit X2 and a second 
input connected to the summer output, which develops 
output bit Y2, by a multiplexer 46 and a 12 -symbol delay 
element 47. The output of delay element 47 is also 
coupled back to its input by multiplexer 46. Output bit 
Y2 of summer 44 is applied as bit 22 to one input of a 
symbol mapper 49, which is shown in more detail in Figure 
5. 

Uncoded bit from precoder 32a is applied to a 
rate 1/2, 4-state, systematic feedback convolution 
encoder 32b for conversion to output bits 2^ and Zq , 
Convolution encoder 32b comprises a signal path 51 for 
applying bit .Y^ directly to a second input of symbol 
mapper 49 as bit and to one input of a modulo -2 summer 
52 . The output of summer 52 is applied through a 
multiplexer 53 to the input of a 12 -symbol delay element 
54, whose output is applied to a third input of symbol 
mapper 4 9 as bit Zq through a second multiplexer 55 

to the input of a second 12-symbol delay element 56. The 
output of delay element 56 is applied to the second input 
of summer 52. The outputs of delay elements 54 and 56 are 
also coupled back to their respective inputs by 
multiplexers 53 and 55. Each of the delay elements 47, 
54 and 56 is clocked at the symbol rate (about 10.76 M 
symbols/sec) . It will be appreciated that each 
subsegment A-L (see Figure 3) will be independently 
processed by precoder 32a and convolution encoder 32b on 
account of the 12 -symbol delay elements which 
characterize their respective operations. 
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Convolution encoder 32b may take various other forms 
from that shown in Figure 4 without departing from the 
invention. For example, the number of encoder states 
may differ from that shown, feedforward architectures may 
be used rather that the disclosed feedback structure and 
non- systematic coding may be employed in either a 
feedback or feedforward arrangement . 

Multiplexers 46, 53 and 55 are provided to allow for 
sync insertion during which times their respective B 
inputs are selected. At all other times the A inputs of 
the multiplexers are selected. Considering operation of 
the circuit when the A input of the multiplexers are 
selected and disregarding, for the moment, the effect of 
precoder 32a, the operation of convolution encoder 32b 
and mapper 49, hereinafter referred to as trellis encoder 
(TE) 60, is illustrated in the table of Figure 6. The 
first column of the Table represents the four possible 
states Qo of the delay elements 56 and 54 of 
convolution encoder 32b at an arbitrary time n. These 
states are 00, 01, 10 and 11. The second column 
represents the possible values of bits for each of 

the states Qi Qq of encoder 32b at time n. The third 
column of the table represents the values of output bits 
^2^1^0 St time n for each combination of bits Y2 and 
encoder states Qi Qq at time n. For example, when 
encoder 32b is in state QiQq = 01, bits Y2 Y^ = 10 
result in output bits Z2 2i Zq = 101. The fourth column 
of the table, labeled R(n), represents the amplitude of 
the symbol provided by symbol mapper 49 (see Figure 5) in 
response to output bits Z2 Zi Zq . Since there are three- 
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output bits, 8 symbol levels (-7, -5, -3, -1, +1, +3, +5 
and -f-7) are provided. Output bits Z2 Zi Zq = 101, for 
example, result in symbol level +3 being generated by 
symbol mapper 49. Finally, the fifth column of the 
table represents the state of encoder 32b at time (n + 
1) . It will be understood that since each of the delay 
elements 54 and 56 is 12 -symbols long, for the symbols of 
each subsegment A-L» the states Qjl Qq of encoder 32b at 
times n and (n + 1) represent successive encoder state 
transitions. 

It will be observed that the 8-level symbols 
developed at the output of mapper 49 are symmetrical 
around the zero level. To facilitate signal acquisition 
in the receiver, it is preferred to offset each symbol by 
a given amount (e.g. +1 unit) to in effect provide a 
pilot component. The symbols and pilot component are 
then applied through a multiplexer 62 to modulator 36 
(see Figure 2) where they are used to modulate a selected 
carrier for transmission in a suppressed carrier VSB form 
as described in the previously mentioned *975 patent- The 
output of mapper 49 is also applied to the input of a RAM 
64, whose output is applied to a second input of 
multiplexer 62. A third input of multiplexer 62 is 
supplied from a source 66 of segment and frame sync 
signals. 

With further reference to symbol mapper 4 9 of Figure 
5, it will be observed that the 8 symbol levels are 
divided into 4 subsets a, b, c, and d, each subset being 
identified by a particular state of output bits Z^ Zq . 
Thus, output bits Z2 Zq = 00 selects symbol subset d, Zn 
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Zq = 01 selects symbol subset c, Zq = 10 selects 
symbol subset b and Z^ Zq = H selects subset a. Within 
each subset, the respective symbol amplitudes differ by a 
magnitude of 8 units. It will also be observed that 
successive symbol levels (-7, -5), (-3, -1), (+1, +3) and 
(+5^ +7) are selected by common states of output bits Z2 
Zi- Thus, for example, output bits Z2 = 00 selects 
both symbol amplitude levels -7 and -5, and so on. Both 
of the foregoing attributes of symbol mapper 4 9 are 
useful in achieving reduced receiver complexity as will 
be described in more detail hereinafter. 

Figure 7 is a state transition diagram for 
convolution encoder 32b derived from the table of Figure 
6 . The diagram illustrates the four states of the 
encoder and the various transitions there between. In 
particular, each state has two parallel branches, each 
extending to the same or another state. The branches are 
labeled with the input bits ¥2 causing the state 
transition and the resulting output R of mapper 49. As 
will be explained in further detail hereinafter, this 
state diagram may be used to design an optimum maximum 
likelihood sequence estimation (MLSE) Viterbi decoder in 
the receiver for recovering estimations of bits Y2 and 
as is well known in the art. 

Figure 8 illustrates the receiver decoding aspects 
of the invention in more detail. The multibit symbol 
values from tuner, demodulator, A/D 40 are applied to a 
first demultiplexer 70 through the first processing path 
comprising comb filter 42 and equalizer 38 and to a 
second demultiplexer 72 through the second processing 
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path comprising equalizer 38. Comb filter 42 comprises a 
feedforward filter including a linear summer 74 and a 12 
symbol delay element 76. As more fully explained in the 
previously mentioned '975 patent, the filter is operable 
for reducing NTSC co- channel interference by subtracting 
from each received symbol, the symbol received 12 symbol 
intervals earlier. Because of the symbol interleaving 
provided in the transmitter, the comb filter 
independently operates on each of the subsegments for 
providing successive combed outputs of the form A^-Aq, 
B3_~Bo, etc. These combed outputs are demultiplexed by 
demultiplexer 70 into 12 separate outputs, each 
corresponding to a respective one of. the subsegments A-L. 
.Each combed subsegment is applied by demultiplexer 70 to 
a respective Viterbi decoder 44A-44L which is operated at 
a rate of 1/12 the symbol clock rate (fs) . Each of the 
decoders 44A-44L provides a pair of output decoded bits 
comprising estimations of input bits Xi X2 , the decoded 
bits being multiplexed into an interleaved bit stream as 
shown in Figure 3 by a multiplexer 78. 

The interleaved symbols from unit 4 0 are also 
demultiplexed by demultiplexer 72 into the 12 separate 
subsegments A-L, each being applied to a respective one 
of the Viterbi decoders 46A-46L. ■ It will thus be seen 
that each of the original data bytes from source 24 are 
processed as a unit by a respective one of the decoders 
46A-46Li. For example, the data byte represented by 
symbols A3 A2 Ai Aq is processed by decoder 46A> and so 
on. The same is of course true for decoders 44A-44L, 
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except that the processed symbols have previously been 
combed by filter 42. 

Each of the decoders 4 6A-46L may comprise a 
substantially identical device operating at the rate of 
fs/12 and programmed according to the state diagram of 
Figure 7 for effecting optimum MLSE Viterbi decoding for 
recovering estimations of bits Y2 and as is well known 
in the art. In particular, each of the decoders 46A-46L 
is programmed to generate 4 branch metrics, typically 
using an appropriately programmed ROM, each representing 
the difference between the received symbol level (i.e. an 
8-10 bit value) and the closest one of the two subset 
levels of each of the symbol subsets a, b, c, and d. 
Figure 9 illustrates a Viterbi decoder manufactured by 
LSI Logic Corp. which may be programmed to perform the 
functions of each of decoders 4 6A-4 6L. The decoder 
comprises a branch metric generator ROM 84 responsive to 
the received symbols for generating and applying 4 branch 
metrics to an add, compare and select (ACS) unit 86. ACS 
unit 86 is bidirectionally coupled to a path metric 
storage memory 88 and also supplies a traceback memory 
90- In general, ACS unit 86 adds the branch metrics 
generated by generator 84 to the previous path metrics 
stored in memory 88 to generate new path metrics, 
compares the path metrics emanating from the same states 
and selects the ones with the lowest path metrics for 
storage. Traceback memory 90, after a number of 
branches have been developed, is operable for selecting a 
surviving path and generating estimations of the bits Y2 
and Yi that would have produced the surviving path. 
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It will be recalled that in the foregoing analysis 
the effect of precoder 32a on the input bit stream had 
been ignored. While the function of the precoder will 
be described in further detail hereinafter, suffice it 
for now to recognize that input bit X2 differs from bit 
¥2 due to the operation of the modulo- 2 precoder. The 
output of each Viterbi decoder 46A-4 6L in Figure 8 
comprises only an estimation of bit Y2, not input but X2 - 
Consequently, a complementary modulo-2 postcoder 48A-48L 
is used in the receiver to recover estimations of input 
bits Xi and X2 from each respective decoder 46A-46L. 
Each postcoder 48A-48L comprises a direct path between 
input bit Y3_ and output bit Xj and a feedforward circuit 
in which output bit Y2 is applied directly to one input 
of a modulo- 2 adder 92 and to a second input of adder 92 
via a one-symbol delay element 94 . The output of adder 
92 "comprises an estimation of input bit X2 • Finally, the 
decoded bits X^, X2 from post coders 4 8A-48L are 
multiplexed into an interleaved bit stream as shown in 
Figure 3 by a multiplexer 96. 

In an alternate embodiment of the invention, each of 
the Viterbi decoders 4 6A-4 6Ii may be replaced by a slicer 
98 as illustrated in Figure 10 to provide a cost reduced 
receiver in cases where the received signal is 
characterized by a relatively high S/N ratio. This is 
frequently the case in cable transmissions which normally 
exhibit a better S/N ratio than terrestrial 
transmissions. A tradeoff is therefore made between TCM 
coding gain and receiver complexity and cost . Referring 
to Figure 10, slicer 98 is characterized by three slice 
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levels (-4, 0 and +4) . A received symbol having a level 
more negative than -4 will be decoded by slicer 98 as 
bits Y2Y1 = 00, a level between -4 and 0 as bits Y2 = 
01, a level between 0 and +4 as bits Y2 Y^ = 10 a level 
more positive than +4 as bits Y2 Y^ = 11. As before, 
bits Y2 Yi are converted to an estimation of bits X2 X^^ 
by a respective postcoder 48A-4 8L. Referring back to 
mapper 4 9 of Figure 5, it will be seen that slicer 98 
effects proper decoding of the received symbols because 
successive symbol levels are represented by common values 
of bits Z2 Z^, as previously mentioned. This embodiment 
of the invention therefore, in effect, implements a 4- 
level transmission and reception system which provides an 
equivalent bit rate as the 8 -level TCM system, but with 
worse S/N perfonnance since the TCM coding gain is not 
realized. 

Referring back to Figure 8, although comb filter 42 
has the desired affect of reducing NTSC co- channel 
interference, it also increases the complexity of 
decoders 44A-44L where optimum MLSE Viterbi decoding is 
used to recover bits and X2 . In particular, an 
optimum MLSE Viterbi decoder must take into account not 
only the state of the encoder, but also the state of 
delay element 76 of comb filter 42 . Since there are 4 
encoder states and 4 possible ways to enter each state 
(i.e. there are 4 possible states of delay element 7S for 
each state of encoder 32b) , an optimum decoder must 
process a 16 -state trellis. In addition, the decoder 
must account for 4 branches entering each state whereas 
only 2 branches enter each encoder state . Such a decoder 



SUBSTITUTE SHEET (RULE 26) 



wo 96/02100 



PCTAJS95/08174 



17 

is illustrated in Figure 11 and, while complex in nature, 
its design is relatively straight forward. In 
particular, while the functionality of the decoder is 
similar to that shown in Figure 9 (the same reference 
numerals are therefore used) , its complexity is greatly 
increased including the requirement to generate 15 branch 
metrics instead of just 4 . The branch metrics represent 
the difference between a received symbol level and each 
of the possible 15 constellation points at the output of 
comb filter 42 (i.e. the linear combination of the 8- 
level symbols provides 15 possible output levels) . 

The table of Figure 12 illustrates a technique 
according to the invention for reducing the complexity, 
and thereby the cost, of the Viterbi decoders 44A-44L 
used to recover bits and X2 from the output of comb 
filter 42. This simplification, which is made possible 
by preceding bit X2 as shown in Figure 4 , is achieved by 
ignoring some of the state information from delay element 
76 of comb filter 42 in constructing the trellis diagram 
forming the basis of the decoder. In particular, as will 
be explained in further detail below, decoding 
simplification is achieved according to this aspect of 
the invention by considering only the information 
identifying the subsets a, b, c and d of the 8 possible 
states of delay element 76 of the comb filter. If the 
output of delay element 76 is represented by reference 
letter V, the combined state of the encoder and channel 
can be represented as (n) Qq (n) ViVq (n) , where subset V^^ 
Vq (n) = subset Zq (^^-D • That is, the state of delay 
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element 76 is represented by the subset of the previous 
symbol . 

Referring now to the table of Figure 12, the first 
column represents the state of the combined encoder and 
channel (using only subset information to represent the 
state of delay element 76) QiQoViVq at time n. As 
shown, there are 8 possible states 0000, 0010, 0100, 
0110, 1001, 1011, 1101 and 1111 (note that in all 
instances Qi = Vq) - These eight states are derived from 
the last two columns of the table of Figure 6 which gives 
the states Qi Qq of encoder 32b and the associated V^^ Vq 
subset of the output V of delay element 76 at an 
arbitrary time (n + 1) . It will be noted that the Vq 
subset at time (n+ 1) is the same as output bits Zq at 
time n (see the third column of the Figure 6 table) . 
Each state QiQo^iVq of the combined encoder and channel 
is listed twice in the table of Figure 12, once for each 
possible value of input bit (see the third column of 

the table) . The fourth column of the table represents 
the subset Z^Zq at time n for each encoder /channel state 
and each value of input bit X^^ . These values are derived 
on the basis of the relationships Zi = X^ and Zq = Qq • 
Both the Vi Vq subset in the first column of the table 
and the Z^Zq subset comprising the fourth column of the 
table are identified by the subset identifiers (a-d) 
shown in mapper 4 9 of Figure 5 in the second and fifth 
columns respectively of the table. 

Referring back to Figure 8, the output of linear 
summer 74 of comb filter 42 applied to each decoder 44A- 
44L is identified by the letter U and comprises the value 
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of a received symbol minus the value of the previous 
symbol. This value is represented in the sixth column of 
the table of Figure 12 as the difference between the Z 
subset Zi Zq and the V subset Vq in terms of the 
subset identifiers (a-d) . Thus, for example, the U 
subset at time n for the first row of the table is (d-d) , 
for the fifth row (c-d) , and so on. In Figure 13 the 
possible values of the U subset are derived by 
substracting each V subset (a, b, c and d) from each Z 
subset (a, b, c and d) . In particular, each possible Z 
subset is identified along the top of the Figure by the 
darkened circles corresponding to the levels of the 
respective subsets. For example, sxibset a comprises 
levels -1 and +7 of the 8 levels, siibset b comprises 
levels -3 and +5, and so on. Likewise, each possible V 
subset is identified along the left-hand margin of the 
Figure. The results of subtracting each V subset from 
each Z subset to derive the U subsets (U = Z-V) are shown 
in the interior of the Figure. For example, the U subset 
(a-a) , see the last row of the table of Figure 12, is 
derived by subtracting the a subset levels -1 and +7 from 
the a subset levels -1 and +7, which gives the three 
levels +8, 0, -8 as shown in the upper left-hand corner 
of Figure 13. Similarly, the U subset (a-b) , see the. 
8th row of the Figure 12 table, is derived by subtracting 
the b subset levels -3 and +5 from the a subset levels -1 
and +7, which gives the three levels +10, +2, -6 as 
shown, and so on. 

Examination of the 16 U siibsets shown in Figure 13 
reveals that each belongs to one of 7 common siibsets 
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hereinafter referred to as cosets. These 7 cosets are 
shown in Figure 14 and identified as cosets A (U subsets 
a-a, b-b, c-c and d-d), Bl (U subsets b-a, c-b and d-c) , 
B2 (U subset a-d) , CI (U subsets c-a and d-b) , C2 (U 
subsets a-c and b-d) , Dl (U subset d-a) and D2 (U subsets 
a-b, b-c and c-d) . The coset for each U subset is also 
shovm in the 7th column of the table of Figure 12 . It 
will be observed that each coset comprises 3 of 15 
possible levels. 

The final column of the table of Figure 12, which 
corresponds to the last two columns of the table of 
Figure 6 , represents the state QiQo^iVo 
encoder/channel at time (n + 1) . The first and last 
columns of the table can now be used to construct a 
trellis state transition diagram for the combined 
encoder/channel as shown in Figure 15. In this Figure, 
Vq has been disregarded since it is redundant with . 
The trellis state transition diagram thus comprises 8 
states, with two branches emanating from each state. 
Each branch is labeled with the input bit X^^ and the 
coset A, Bl, B2, CI, C2, Dl and D2 associated with the 
respective transition. The trellis diagram of Figure 15 
can now be used to provide the basis of a reduced 
complexity Viterbi decoder (for each of decoders 44A-44L) 
for estimating input bit from the output U of summer 
74 of comb filter 42. This decoder, which comprises an 
alternate embodiment of the optimum Viterbi decoder of 
Figure 11, may take the form of the Viterbi decoder 
illustrated in Figure 16 . The apparatus used to 
implement this Viterbi decoder may be similar to that 
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used in the decoder of Figures 9 and 11 and thus 
comprises a branch metric generator 84, an ACS unit 86, a 
path metric storage memo3ry 88 and a traceback memory 90. 
In the case of the decoder of Figure 16, branch metric 
generator 84 is programmed to generate seven branch 
metrics each representing the squared Euclidean distance 
between the symbol level U at the output of summer 74 of 
comb filter 42 and the nearest one of the 3 valid levels 
of each of the 7 cosets A, Bl, B2 , CI, C2 , Dl and D2 . 
For example, assuming a level U = (-6), the seven branch 
metrics would be derived as follows: A= 2^ = 4; Bl = 4^ = 
16; b2 = 42 = 16; CI = 2^ = 4 ; C2 = 22 = 4 ; Dl = 0 and D2 
= 0. Based on these branch metrics and the trellis 
diagram of Figure 15, the decoder provides an estimation 
of bit Xi and the associated coset identification, which 
are known from the surviving path decisions made by the 
decoder - 

It is still, however, necessary to provide an 
estimation of input bit X2 and this may be done in 
response to the coset information provided by the Viterbi 
decoder of Figure 16. The ability to so estimate bit X2 
is facilitated by providing precoder 32a in the path of 
input bit X2 in Figure 4. In particular, it will be seen 
that precoder 32a is configured such that whenever input 
bit X2 (n) =1, the corresponding output bit Y2 (n) of 
the precoder is different from the previous output bit Y2 
(n-l) . That is, if Y2 (n) se Y2(n-1), then X2 (n) = 1. 
Also, if X2(n) = 0, then the corresponding output bit 
Y2 (n) will be equal to the previous output bit Y2 (n-1) . 
That is, if Y2 (n) = Y2 (n-l) , then X2 (n) ^ 0. Also, with 
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reference to mapper 4 9 of Figure 5, it will be observed 
that a positive level symbol is provided when Z2 (i.e. 
Y2) =1 and a negative level symbol is provided when Z2 = 
Y2 = 0. 

The foregoing characteristics are used to estimate 
bit X2 as shown in Figure 17. The symbol level U at the 
output of summer 74 of comb filter 42 is applied through 
a delay 100 (chosen to match the delay of Viterbi 
decoders 44A-44L) to one input of a plurality, i.e. 7, 
of slicers 102. The coset identification signal at the 
output of Viterbi decoder 44A-44L is applied to the 
second input of slicer 102. An estimation of bit X2 is 
developed by slicer 102 by determining whether the U 
symbol level from comb filter 42 is closer to one of the 
outer levels (e.g. levels +8 or -8 of coset A) of the 
coset A, Bl, B2, CI, C2 , Dl or D2 identified by the 
respective Viterbi decoder 44A-44L, in which case bit X2 
is decoded as a 1, or whether it is closer to the 
intermediate level (e.g. level 0 of coset A) of the 
identified coset level, in which case bit X2 is decoded 
as a 0 • The foregoing is based on the fact that the 
positive outer level (e.g. +8 of coset A) of each of the 
cosets results only when successive Y2 bits at the output 
of precoder 32a are characterized by the values Y2 (n) = 
1 and Y2(n-1) = 0, the negative outer level (e.g. -8 of 
coset A) of each coset only when successive Y2 bits have 
the values Y2 (n) = 0 and Y2(n-1) = 1 and the intermediate 
level (e.g. 0 of coset A) of each coset only when 
successive Y2 bits have values Y2 (n) = 1 and Y2(n-l) = 1 
or Y2 (n) = 0 and Y2 (n-l) =0. In the two former cases 
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X2(n) = 1 [since Y2 (n) ae ¥2(11-1)] and in the latter case 
X2(n) = 0 [since Y2 (n) = Y2(n-1)3. 

Finally, it will be \inderstood that the inclusion of 
precoder 32a in the path of input bit X2 requires the 
incorporation of a complementary postcoder 104 in the 
path of estimated bit X2 when an optimal MLSE Viterbi 
decoder is used to process the output of comb filter 42 
as shown in Figure 11. A complementary postcoder is not 
required in the case of the circuit of Figure 17 since 
estimated bit X2 is directly produced. 

As previously described, the data provided by source 
24 is preferably arranged in successive data frames, each 
comprising, a plurality of data segments of 684 symbols, 
although the following aspects of the invention are 
equally applicable to arrangements having different 
numbers of data segments per frame and different numbers 
of symbols per data segment. It is further desirable to 
incorporate a frame sync signal, which may comprise one 
or more pseudo-random sequences, in the first data 
segment of each frame and a data segment sync signal in 
the first four symbol positions of each data segment. 
Referring back to Figure 4, the frame and segment sync 
signals are inserted at the appropriate times into the 
data stream at the output of multiplexer 62 by frame and 
data segment sync generator 66. During these intervals, 
the B input of multiplexer 46 of precoder 32a and the B 
inputs of multiplexers 53 and 55 of convolution encoder 
32b are selected. -Also, the last 12 symbols of the last 
data segment of each frame are read into memory 64 and 
copied into the last 12 symbol intervals of the frame 
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sync segment at the output of multiplexer 62. As will be 
explained in further detail hereinafter, the foregoing 
orovisions are effective to insure that in the receiver 
symbols from each of the subsegments A-L (see Figure 3) 
are only processed with symbols from the same subsegment . 

More specifically/ during the segment sync interval 
4 predetermined sync symbols Sq, Si, S2. and S3 are 
inserted into the data stream by generator 66 and 
multiplexer 62 while, at the same time, input data from 
source 24 is temporarily suspended. Also, since the 
outputs of delay elements 48, 54 and 56 are being fed- 
back to their respective inputs, each of the delay 
elements will be characterized as shown in Figure 18 
immediately following the segment sync interval, wherein 
the state of the delay elements is defined by a symbol 
from subsegment E. The 

composite ' signal in the vicinity of the segment sync 
signal Sq, Si, S2 and S3 is illustrated in Figure 19, in 
which the data segment containing the sync signal occurs 
at time n and the preceding and following segments occur 
at times (n-1) and (n +1) respectively. In connection 
with this Figure, it will be noted that subsegment 
integrity is maintained (all symbols from the same 
subsegment are spaced from each other by 12 symbol 
intervals) , despite the incorporation of the sync symbols 
into the composite data stream. 

Figure 20 shows an embodiment of comb filter 42 of 
Figure 8 modified for operation in accordance with the 
sync insertion aspects of the invention. The 
modification comprises the provision of a multiplexer 110 
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having an A input for directly receiving the output of 
the comb filter and a B input for receiving the output of 
a summer 112. One input of summer 112 is connected • 
directly to the output of the comb filter while its 
second input is connected to the comb filter output by a 
12-symbol delay element 114. -The B input of multiplexer 
110 is selected during symbol intervals 13-16 (i.e. the 
sync interval delayed by 12 symbol clocks) and otherwise 
the A input is selected. 

In operation, the output of comb filter 42 during 
the sync interval comprises: 

50 - A (n-1) 

51 - B (n-1) 
82 - C (n-1) 
S3 - D (n-1) 

This information, which is applied to the decoder 
via the A input of multiplexer 110, does not represent 
meaningful data and is therefore ignored by the decoder. 
However, beginning with the next symbol in the data 
segment occurring at time n (i.e. a symbol from 
subsegment E) , symbols from the same subsegments are 
properly combed together and provided to the decoder via 
the A input of multiplexer 110. During the first 4 
symbols of the data segment occurring at time (n +1) the 
B input of multiplexer 110 is selected. The output of 
comb filter 42 during this period is: 

A (n+1) - So 

B (n+1) - Si 

C (n+1) - S2 

D (n+1) - S3 
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These values are combined in summer 112 with the 4 
outputs of the comb filter during the sync interval 
stored in delay 114 to provide the 4 successive outputs 
A{n+1) - A(n-l), B{n+l) - B(n-l), C(n+1) - C(n-l) and 
D(n+1) -D (n-1) . It will be noted that each output 
represents combed data symbols from the same subsegment 
as desired. Thereafter, the A input of multiplexer 110 
is again selected and normal processing continues. 

Figure 21 shows an embodiment of the post coders used 
in the receiver of the invention, see, for example, 
postcoders 48A-48L of Figure 8 and 10, modified for 
operation in accordance with the sync insertion aspects 
of the invention. The modified postcoder, which 
comprises a modulo adder 120 and a feedforward delay 122, 
further includes a multiplexer 124 for coupling the 
output of delay 122 back to its input during the sync 
interval and otherwise applying the postcoder input 
signal to an input of adder 120 through delay 122. As a 
result, after the sync interval during which the output 
of the postcoder is ignored, each of the modified 
postcoder 4 8A-4 8L will have stored in its respective 
delay 122 the symbol from the subsegment with which it is 
associated as desired. 

Frame sync insertion and processing is effected much 
in the same manner as described above in connection with 
data segment sync. More specifically, during the frame 
sync interval, i.e. the first data segment of each frame, 
generator 66 and multiplexer 62 are initially operated 
for inserting frame sync symbols VQ-Vg?! into the first 
672 symbol positions of the frame sync segment Sq as 
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shown in Figure 22. The last 12 symbols of the frame 
sync segment are inserted into the data stream by RAM 64 
and comprise the last 12 symbols of the last data segment 
S3 12 of the previous frame (which had previously been 
written into RAM 64) . Also, since the B inputs of 
multiplexers 46, 53 and 55 are selected during the frame 
sync interval, delay elements 48, 54 and 56 will assume 
the condition shown in Figure 18 at the end of the 
segment sync interval of the next data segment Si, which 
will then be formed as previously described and as shown 

in Figure 22 . 

The circuits of Figures 20 and 21 operate as 
previously described to insure that symbols from each of 
the subsegments A-L are processed with symbols only of 
the same subsegment . The outputs of the two circuits 
during the frame sync segment Sq do not represent 
meaningful data and are therefore ignored during 
subsequent processing. 

As mentioned previously, the system of the invention 
may be utilized with different mapping constellations to 
provide, for example, increased bit rates and with 
different modulation schemes such as QAM. Figure 23 
illustrates an application of the invention to a system 
wherein each symbol represents 3 bits instead of 2 bits 
as previously described. As illustrated in the drawing 3 
input data bits Xi, X2 and X3 are provided at the symbol 
rate, bits X3 and X2 being converted by a modulo 4 
precoder 32a', which includes a modulo 4 combiner 44", 
to bits Y3 and ¥2 for application as bits Z3 and Z2 to a 
16-level symbol mapper 49'. Data bit Xi is applied as 
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bit Zi to a third input of mapper 49' and to convolution 
encoder 32b which develops bit Zq for application to the 
fourth input of mapper 49'. As in the previously 
described embodiment, bits Zi Zq identify subsets a, b, 
c, and d, each of which coir^jrises 4 symbol levels. 
Also, within each subset the respective symbol amplitudes 
differ by a magnitude of 8 units and successive symbol 
levels (e.g. -15, -13) are selected by common states of 
bits Z3 Z2 Zi- The signal generated by the circuit of 
Figure 23 may therefore be decoded using the techniques 
previously described. In this example, an optimum MLSE 
decoder (i.e. one that does not take into account the 
precoder and is used to decode the output of the comb 
filter) would have 8 times the number of states that the 
encoder has. The inclusion of the modulo-4 precoder 
allows the decoder to operate on a trellis that has only 
twice as many states as the encoder and still decode the 
uncoded bits without error propagation. 

Figures 24A and 24B illustrate the application of 
the invention to a QAM modulator. As shown in Figure 
24A, 3 inputs bits Xi, X2 and X3 are provided, bits X3 
and X2 being independently preceded by respective modulo 
- 2 precoders 32a' • and 32a''' to provide output bits Z3 
and Z2 and bit Xi being supplied to convolution encoder 
32b for generating output bits Zi and Zq . Output bits Z3 
Z2 Zi Zq. are applied to a symbol mapper 49' ■ for 
generating 16 quadrature related symbols (see Figure 24B) 
belonging to one of the subsets a - d for application to 
a QAM modulator 36'. In connection with the foregoing, 
it will again be observed that bits Zi Zq identify the 
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respective symbol subsets a - d. Optimum decoding 
without the precoders would require a decoder having 2- 
8 times the number of states that the encoder has. With 
the precoders, the decoder would only have twice the 

number of states . 

Receivers for the systems of Figures 23 and 24 may 
take the form generally illustrated in Figure 8. In the 
case of the system of Figure 23, a modulo 4 postcoder 
4 8A', including a modulo 4 combiner 92', as shown in 
Figure 2 5A would replace each modulo 2 postcoder. 48A 
and, in the case of the system of Figures 24A and B, a 
pair of modulo 2 postcoders 48A' • and 48A' ' ■ as shown in 
Figure 25B would replace each modulo 2 postcoder 48A. 

It is recognized that numerous changes in the 
described embodiments of the invention will be apparent 
to those skilled in the art without departing from its 
true spirit and scope. The invention is to be limited 
only as defined in the claims. 
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1. A data receiver including means for developing a 
received data signal including a plurality of regularly 
spaced symbols each representing 3 or more bits Zq, Z^, 
- Zn produced by coding 2 or more data bits Xj, Xj, - Xj, 
with a coder including a multi-state convolution encoder, 
a multi-state linear filter for reducing co-channel 
interference characterizing said received data signal, 
and decoding means including a Viterbi decoder for 
estimating data bits X^, X^, - X„ from the linearly 
filtered data signal in response to a .determination of 
the most likely sequence of transitions between a 
combination of the states of said convolution encoder and 
at least a partial representation of the states of said 

linear filter- 

2 . The receiver of claim 1 wherein said linear 
filter includes an M-state linear filter and said linear 
filter includes an M-state linear filter and said 
received data signal includes M groups of individually 
coded interleaved symbols, said receiver including M of 
said decoding means and means for applying each of said M 
groups of symbols to a respective one of said M decoding 
means . 

3 . The receiver of claim 1 or 2 including a second 
Viterbi decoder means for estimating data bits X^, Xj - Xj, 
from said received data signal in response to a 
determination of the most likely sequence of transitions 
between the states of said convolution encoder. 

4 . The receiver of claim 1 wherein said 3 or more 
bits Zo, Zi, Zj, - Zk include bits Zo, Z^, Z^ produced by 
coding data bits Xj, Xj and wherein said decoding means 
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includes means for estimating daca bits X,, X, from the 
linear filtered data signal in response to a 
determination of the most likely sequence of transitions 
between a combination of the states of said convolution 
encoder and a subset of the states of said linear filter. 

5. The receiver of claim 4 wherein the states of 
said linear filter are representable by a 3 -bit value and 
wherein said subset includes one of the bits of said 3- 
bit value. 

6. The receiver of claim 4 or 5 wherein said 
decoding means includes a Viterbi decoder for estimating 
data bit X, and for identifying one of a plurality of 
cosets of discrete values representing a predetermined 
number of anticipated outputs of said linear filter. 

7. The receiver of claim 6 including means 
responsive to said linearly filtered signal and to said 
identified coset of discrete values for estimating data 
bit Xa- 

8. The receiver of claim 7 wherein the output of 
said linear filter is representable by 15 different 
•discrete values and wherein each of said plurality of 
cosets includes a different combination of 3 of said 15 
different discrete values, said means for estimating data 
bit X2 including means for determining whether the signal 
level at the output of said linear filter is closer to 
the intermediate value or to one of the outer values of 
the 3 discrete values of the identified coset. 

9. A data receiver including means for developing a 
received data signal including a plurality of regularly 
spaced symbols, each of said symbols representing 3 -bits 
Zo, Z„ Za produced by coding a first data bit X^ using a 
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multi- State convolution coder to derive bits Zo, Z, and a 
second data bit using a Tnodulo-2 precoder to derive bit 
Z2, wherein bits Zo, identify respective subsets (a-d) 
of said symbols, each of said subsets (a-d) including two 
equally spaced symbol values, and bits Zj, Z2 identify 
respective pairs of said symbols each including two 
adjacent symbol values, a multi- state linear filter for 
reducing NTSC co- channel interference defining said 
received data signal, first decoder means including a 
Viterbi decoder for estimating data bits Xj., X2 from said 
linearly filtered data signal in response to a 
determination of the most likely sequence of transitions 
between a combination of the states of said convolution 
coder and a subset of the states of said linear filter, 
second decoder means for estimating data bits Xi, X2 from 
said received data signal, and selection means for 
selecting said first decoder means or said second decoder 
means for providing said estimated data bits X^, Xj . 

10 The receiver of claim 9 wherein said Viterbi 
decoder includes a branch metric generator for generating 
a plurality of branch metrics each representing the 
distance between the output of the linear filter and the 
closest value of a respective coset of 3 values, each of 
said cosets representing the linear difference between 
the values of the symbols of two said subsets. 

11. The receiver of claim 10 wherein said Viterbi 
decoder includes means for estimating data bit X^ and for 
identifying an associated one of said cosets. 

12. The receiver of claim 11 including means 
responsive to said linearly filtered signal and to said 
identified coset for estimating data bit Xj. 
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13 The receiver of claim 12 wherein said means for 
estimating data bit X2 includes means for determining 
whether the signal level at the output of said linear 
filter is closer to the intermediate value or to one of 
the outer values of the 3 values of said identified 
coset . 

14 . The receiver of any of claims 9 through 13 
wherein the states of said linear filter are represented 
by a 3-bit value Vq, V^, and wherein said subset 
includes bit of said 3 -bit value. 

15. The receiver of any of claims 9 through 13 
wherein said first and second decoder means include a 
Viterbi decoder means responsive to said selection means 
for assuming a configuration for performing the function 
of said first decoder means or a configuration for 
performing the function of said second decoder means. 

16 . A Viterbi decoder for decoding a signal 
generated by a multi- state channel, said Viterbi decoder 
including a branch metric generator responsive to said 
received signal for generating plurality of branch 
metrics, and means responsive to said branch metrics for 
determining the most likely sequence of transitions 
between a combination of the states of said convolution 
encoder and a subset of the states of said channel. 

17. The decoder of claim 16 wherein each of said 
branch metrics represents the distance between the value 
of said received signal and the closest value of said 
received signal and the closest value of a respective 
coset of N values. 

18. The decoder of claim 17 wherein said 
determining means includes means for directly estimating 
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a first data component of said received signal and for 
identifying an associated one of said cosets. 

19. The decoder of claim 18 including means 
responsive to said received signal to said identified 
coset for estimating a second data component thereof. 

20. The decoder of claim 19 wherein each of said 
cosets includes 3 data values and wherein said means for 
estimating said second data component includes means for 
determining whether the level of said received signal is 
closer to the intermediate value or to one of the outer 
values of the 3 values of said identified coset. 
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